MEMORANDUM August 21, 2015 
TO: Board Members 


FROM: Terry B. Grier, Ed.D. 
Superintendent of Schools 


SUBJECT: NEW DIGITAL ENERGY GAME PROGRAM EVALUATION, 2014-2015 
CONTACT: Carla Stevens, (713) 556-6700 


The New Digital Energy Game (NDE), sponsored by Chevron, in collaboration with Tietronix 
Software, Inc. and the Houston Independent School District (HISD), was implemented for the 
fifth consecutive year in HISD’s middle and high schools. NDE combines aspects of strategy, 
construction, and game management, requiring players to build energy companies, gain 
dominant market share, and meet the needs of cities throughout the United States. Cooperative 
teams of students play against artificial intelligence, competing across three levels of difficulty. 
Variations in difficulty are incorporated into lessons that students must master in order to open 
options within the game. 


Paired t-test analysis of a student sample yielded a statistically significant increase in students’ 
overall interest rating in science from pre- to post survey, and a decrease in their overall attitude 
rating over the same time period. Propensity score, nearest neighbor matching yielded a higher 
percent of items correct on the mid-year District Level Assessment (DLA) in science for NDE 
students compared to students who did not participate in the program. The difference between 
the groups was statistically significant, suggesting that the NDE program had a positive effect 
on students’ DLA science performance. 


Administrative Response: The HISD Strategic Partnership Department will continue to 
coordinate with Chevron to support implementation of prekindergarten through 12 engineering 
education activities. These efforts will also serve as a resource to help teachers generate future 
engineers, scientist, geologist, doctors, and other STEM career opportunities for HISD students. 


Should you have any questions or require any further information, please contact me or Carla 
Stevens in the Department of Research and Accountability, at 713-556-6700. 


“S ae BG 


TBG/CS:vh 
cc: Superintendent's Direct Reports Lance Menster 
Chief School Officers Anne Wolfe 
School Support Officers Rose Adams 
Caleen Allen Hortense Campbell 


Lucy Bremond Teresa Phillips 
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The effect of the New Digital Energy game on students’ interest, attitude, and 
science achievement using propensity score matching, 2014-2015 


By Venita Holmes, Dr.P.H. 


During the 2014-2015 academic year, 1,148 students registered and 820 elementary, middle, and high- 
school students played the New Digital Energy Game (NDE) game in the Houston Independent School 
District (HISD). The NDE game was designed to enrich students’ science experiences in order to stimulate 
their interests and attitudes to pursue science careers. The game was funded by Chevron Corporation and 
developed by Tietronix, Inc. Paired t-test analysis of a student sample yielded a statistically significant 
increase in students’ overall interest rating in science from pre- to post survey, and a decrease in their 
overall attitude rating over the same time period. Propensity score, nearest neighbor matching yielded a 
higher percent of items correct on the mid-year District Level Assessment (DLA) in science for NDE 
students compared to students who did not participate in the program. The difference between the groups 
was Statistically significant, suggesting that the NDE program had a positive effect on students’ DLA 


science performance. 


Notable highly statistically significant intercorrelations were as students’ post- 


interest and post-attitude ratings increased, their science DLA results increased. 


Background 


U.S. educators have persistently sought to 
improve academic achievement for low-performing 
students through reform laws such as No Child Left 
Behind (U.S. Department of Education, 2001; 
Education Week, 2011). However, the lack of 
substantial academic progress for students over the 
years (Phillips, 2014) has led to the emergence of 
new technologies to support student’s academic 
success. A shift in education from a didactic model 
of instruction to a constructivist model that 
emphasizes students as active learners has played a 
pivotal role in the development of these 
technologies (Jong, Shang, Lee, F., & Lee, J., 
2008). 

Much attention has been devoted toward 
understanding the cognitive, social, and cultural 
features in computer games and their educational 
benefits to students (Gee, 2005; Squire, 2005). 
Research has shown that computer games that 
provide instruction have the potential to stimulate 
the academic environment and increase student’s 
awareness and knowledge of phenomenon to solve 
problems (Honey & Hilton, 2010; Jones, 1996; 


Mundie, 2011; Owston, 2009; Yang, Kun, & 
Chein, 2010). “Fun and enjoyment are important in 
the process of learning as learners can be more 
relaxed, motivated and willing to learn” (Jong, et 
al., p., 54, 2008). Electronic gaming has been 
considered instrumental toward engaging students 
in critical thinking necessary to apply subject 
matter to "real-life" experiences (Curriculum 
Review, 2009). Digital games may encourage 
students to ask unanswered questions and teach 
them to use problem-solving techniques. 

Brendzel (2004) pointed out that “games provide 
a natural motivation” (p. 32) to apply good 
teaching strategies to help build science concepts. 
Contest can further stimulate students’ drive to 
compete by improving their comprehension skills 
through a format that may be more interesting to 
students. Interested students may be more engaged 
in activities that they value (Bulunuz & Jarret, 
2009). Ogunkola (2011) found that increased 
attitudes and interest in an activity is correlated 
with good study habits and engagement in school. 
Disengaged students are more likely to have poor 
attendance and more likely to drop out of school 
(Balfanz, Herzog, & MaclIver, 2006). 
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Figure 1. Number of students who registered and played the NDE game, 2010-2011 through 2014-2015 


NDE game tasks accommodated all academic 
levels by combining strategy, construction, and 
game management, while requiring players to build 
energy companies and gain dominant market share 
to meet the needs of U.S. cities. Students played as 
teams against artificial intelligence, competing 
across three levels of difficulty. Variations in 
difficulty were incorporated into lessons that 
students had to master to open options within the 
game. Lessons and questions prompted students to 
game-play decisions that required understanding of 
physics, chemistry, earth science, and math 
concepts. 

To that end, this study measured the effects of 
the NDE game on students’ science achievement. 
In addition, the study measured _ student’s 
perceptions toward increasing their interests and 
attitudes about science. 


Methods 
All Student Participants 

All HISD science teachers were invited to recruit 
students to participate in the NDE game during the 
fall of 2015. Teachers were required to register. 
Teachers of winning teams had the opportunity to 
attend innovative professional development in 
science. 

Figure 1 presents the number of students who 
registered along with the number of students who 
played the game over the past five academic years. 
Twenty-two schools and 29 teachers were 
represented in the 2012-2013 data; whereas, 16 
schools were represented in the 2013-2014 data. 


By the 2014-2015 academic year, a total of 22 
teachers were reflected in the data at 17 schools. 
(See Appendix A for the 2014-2015 NDE 
schools.) Student registration steadily increased 
from 2010-2011 to 2012-2013 and _ steadily 
decreased from that point to 2014-2015. At the 
same time, students playing the game dropped by 
20.6 percent from the 2012-2013 to the 2013-2014 
academic year, and increased by 26.7% from the 
2013-2014 to the 2014-2015 academic year. 


Selected Student Sample 

The study sample was selected based on having 
completed the pre- and post-science energy interest 
and attitude surveys. Among the 820 students who 
played the game, the student identification numbers 
of 369 students were captured in the survey data. 
The number of students who registered and 
participated in the game by school can be found in 
Appendix B. 


Table 1 presents a profile of the 369 NDE 
students whose data were analyzed in this program 
evaluation. The majority of the NDE student 
sample was female (50.4%) and economically 
disadvantaged (57.5%). Comparatively, the District 
had a_ higher percentage of economically 
disadvantaged (75.5%), at-risk (66.4%), limited 
English proficient (LEP) (29.9%), and special 
education students (7.5%) than the NDE student 
sample. 


Table 1: NDE Student Sample vs. HISD, 2014-2015 


NDE Student Sample" HISD* 

Gender % % 

Male 49.6 51.0 

Female 50.4 49.0 
Eco. Disadv. 57.5 75.5 
At Risk 36.9 66.4 
G/T 28.5 15.4 
LEP 10.8 29.9 
Spec. Ed. 5.4 7.5 


*2014-2015 HISD Facts and Figures 
‘NDE student sample = 369 


Measures 

Students’ interest and attitude were assessed 
using a web-based survey format that was accessed 
at the game site. The instruments included a 10- 
item science interest survey and a 10-item science 
attitude survey. The survey items measuring 
interest and attitude had good internal consistency, 
with a Cronbach alpha coefficient of .81 and .86, 
respectively. While students were encouraged to 
complete the instruments, no incentives were 
offered for completion. Interest was rated using a 
Likert-type scale: very often - 4; regularly - 3; 
sometimes - 2; and never or hardly ever - 1. 
Attitude was rated strongly agree - 4; agree - 3, 
disagree - 2, and strongly disagree - 1. 

Academic performance was measured using 
District Level Assessment (DLA) data in science. 
The DLA was administered in February 2015. The 
timeframe of the science DLA administration was 
consistent with when the NDE program ended in 
February 2015. Propensity score, nearest neighbor 
matching was used to estimate the probability of 
being exposed to treatment given a set of observed 
variables and taking into consideration science 
DLA scores. Students selected for the comparison 
group attended the same schools as NDE students. 
Propensity score matching is a common practice in 
social science to make causal inferences based on 
observational data (Cohen, 1988). NDE students 
who participated in the survey study sample were 
used in the propensity score matching. Gender, 
economic status, LEP, G/T, special education 
program status, and at risk were matching criteria. 
Bias analysis (Appendix C) was conducted and 
yielded a comparison sample in that the treated and 
untreated shared the same characteristics; thus, the 
selection bias had been mitigated in the “new” 
sample (Austin, Grootendorst, and Anderson, 
2007). Based on matching criteria, Stata selected 
193 NDE students and the 46,362 students in the 
comparison group for the model (Appendix D). 

Descriptive statistics were presented on the 
measures. Comments about students’ experiences 
were summarized. Correlation analysis was 
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conducted to determine whether there was a 
relationship between students’ interest, attitude, 
and DLA science performance. 


What was the impact of the NDE game on 


students’ science interest and attitude? 


Data were gathered on NDE student’s science 
interest and attitude before and after participation 
in the game during the 2014-2015 academic year. 
The overall mean along with the mean ratings by 
survey item are presented. Additional paired t-test 
analyses include standard deviations, t-statistics, 
and p-values. 


Science Interest Survey Results 

Figure 2 show a statistically significant 
increase in the students’ overall science interest 
mean rating from pre- (M = 2.24, SD = .666) to 
post-survey (M = 2.31, SD = .699), t (368) = 2.55, 
p = .01. Item-by-item interest ratings can be found 
in Figure 3. Among the 10 items measuring 
interest, there were significant increases in mean 
ratings on three of the items: “the extent that 
students read books or magazines about science 
outside of school” (p < .05), “participated in after- 
school science activities” (p < .001), and “attended 
science study groups” (p < .05). Other increases in 
interest were noted on the items reflecting that 
students “talked to their teacher about science,” 
“participated in science competitions,” “talked to 
their friends about science,” “visited websites 
about science,” and “watched news on TV that 
involved science concepts.” There was a slight 


decrease in ratings on items measuring that 
“students had fun when learning science” and 
science 


“talked about 
members.” 


to parents or family 


Interest Attitude 


=Pre @ Post 


Figure 2: Overall mean interest and attitude ratings from 
pretest to posttest 
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Had fun when learning science. 

Talked to my teacher about science. 

Participated in science competitions. 

Talked to my friends about science. 

**Attended science study groups. 

Visited websites about science. 

***Participated in after-school science activities. 
Talked about science to parents or family members. 


Watched news on TV that involved science concepts. 


*Read books or magazines about science outside of 
school. 


Figure 3: Pre- and post-survey ratings of NDE students on items measuring interest in science (4-point Likert-type scale) 
Note: *Statistical significance at p < .05; **p < .01; ***p < .001 


Science Attitude Survey Results 
Relative to attitude about science, a decrease in _—‘ Item-by-item pre-and post-attitude ratings can be 


students’ overall mean rating was noted from pre- _— found in Figure 4. Ratings increased on the item “I 
(M = 3.05, SD = .717) to post-survey (M = 2.98, would like to have a career involving science” 
SD = .826), t (326) = -1.77, p = .08. from pre- (M = 2.72) to post-survey (M = 2.74). 


However, the positive change was not statistically 
significant at p < .05. 


I would like to have a career involving 
science. 


I study science because I know it is useful 
to me. 

** Making an effort to study science is 

worth it because it will help me get into.. 
Studying science is worthwhile because it 

will improve my career opportunities 

*Science energy game will help me learn 

difficult science theories that I may not... 


™ Pre 


= Post 


1 like studying science. 
Science will help me become successful. 


*T am interested in learning more science. 


- 050 1.00 1.50 2.00 2.50 3.00 3.50 


Figure 4: Pre- and post-survey ratings of NDE students on items measuring attitude about science (4-point Likert-type scale) 
Note: *Statistical significance at p < .05; **p < .01 
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Table 2: T-test Analysis based on the Percentage of Items Correct on Science District Level Assessments, spring 
2015 


n Mean Mean Diff. 
NDE Sample 193 66.8 7.8 
Matched Student Sample 46,362 58.9 
*p < .05 


Statistically significant decreases in mean ratings 
from pre- to post-survey were noted on the 
following items: “making an effort to study science 
is worth it because it will help me get into college” 
(p < .01), the “science energy game will help me 
learn difficult science theories that I may not 
understand without seeing in the game” (p < .05), 
and “I am interested in learning more science” (p < 
.05). 


What was the effect of the NDE game on 


students’ science achievement? 


Propensity score, nearest neighbor matching was 
used to determine the impact of the NDE program 
on NDE student’s 2015 District Level Assessments 
(DLAs) science scores. The statistical model used 
in the analysis controlled for economic status, 
grade level, gender, special education program 
status, and gifted/talented identification, at risk, 
and limited English proficient. Matched students 
were enrolled in the same schools as NDE students, 
and yielded a sample of 193 NDE students and 
46,362 comparison-group students in the model. 

Table 2 presents the academic performance of 
the NDE sample and the matched sample on the 
science DLA (February 2015 administration). The 
NDE student sample had a higher percentage of 
items correct (66.8%) due to participation in the 
program compared to students who did not 
participate in the program (58.9%). The difference 


between the groups was statistically significant (t = 
3.77, p < .001). The results from the model 
suggested that the NDE program had a positive 
effect on students’ DLA science performance. 


What were the associations between science 


interest, attitude, and DLA science results? 


Table 3 depicts means and standard deviations 
along with intercorrelations of the measures for the 
NDE student sample. There were strong positive 
associations between students’ ratings on the 
interest, attitude, and science DLA. A notable 
finding was that as students’ post-interest and post- 
attitude ratings increased, their science DLA results 
increased. These intercorrelations were highly 
statistically significant at p < .01. 


Discussion 


The New Digital Energy game was designed to 
increase student’s understanding of science 
concepts by introducing game construction and 
management strategies. Students worked as teams 
to build energy companies, gain dominant market 
share, and meet the needs of cities throughout the 
United States. Students played against artificial 
intelligence to answer challenging questions about 
science energy. An objective of the game was to 
increase student’s interest and attitude about 
science as they increased their knowledge of 


Table 3: Summary of Intercorrelations, Means, and Standard Deviations for Scores on the Pre- and Post-Science Interest, Attitude, 


and Science DLA Measures for NDE Sample, 2015 (n = 194) 


1 2 3 4 5 

1. Pre-Interest - 

2. Post-Interest 784" - 

3. Pre-Attitude 652" 592" 2 

4, Post-Attitude 585" .698"" 701" - 

5. Science DLA 177* 235%" 167* .328** 7 
Mean 2.09 2.21 2.94 2.81 66.85 
Std. Deviation 63 .76 75 87 18.67 


* Correlation is significant at the p < .05 level (2-tailed test) 
**Correlation is significant p <.01 level (2-tailed) 


science concepts. 

There were several limitations to the study. First, 
given the team format, the study lacked data on 
actual time spent by individual students playing the 
game. Thus, differential effects of student’s 
exposure to the game on their DLA performance, 
interest, and attitude could not be assessed. 
Another limitation was the lack of a comparison 
group to assess the impact of the game on students’ 
interests and attitudes. While a pre- post-test design 
provided an alternative method for comparing 
students, threats to validity could still exist relative 
to knowing whether other external factors 
influenced the changes in student outcomes over 
time (Boyd, 2002). In consideration of these 
threats, propensity score matching was used to 
reduce bias and assess whether participation in the 
game had an effect on _ students’ science 
achievement while controlling for key 
demographic characteristics. The findings included 
a Statistically significant difference between the 
NDE student sample’s DLA science performance 
and the performance of the control group on the 
assessment. Finally, the lack of data at the four 
survey data collection points on all students who 
played the game resulted in substantial attrition. 
(The number of students who played the game was 
substantially lower than the number of students 
who initially registered (820 out of 1,148 students). 
Even less students were included in the final 
analysis of the survey data (369). The recruitment 
process relied on teacher commitment, which may 
have contributed to attrition of the initial group of 
student participants. 

For the current year program evaluation, positive 
correlations were found between student’s interest, 
attitude, and DLA science scores. In addition, as 
student’s science DLAs scores increased, there was 
a highly statistically significant increase in their 
post-interest and post-attitude perceptions about 
science. This finding was consistent with the 
expectation that a positive relationship exists 
between interests and achievement (Brendzel, 
2004). 

This study offers promising results regarding 
influencing student’s perceptions about science and 
science achievement. Moreover, what may be more 
important is that students were exposed to science 
material needed to enhance their understanding of 
science concepts (Wainwright & Linebarger, 
2006). These experiences may be, particularly, 
valuable to students who, otherwise, could not 
afford them. Teachers were provided the 
opportunity to attend workshops to stay abreast of 
innovative trends and best practices in science. 
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Future research may explore other innovative 
strategies for improving student behavior and 
academic achievement in science. This may be, 
particularly, important given the achievement gaps 
that exist for low-performing students in the 
targeted content area (U.S. Department of 
Education, 2001; Education Week, 2011, Phillips, 
2014. 
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Appendix A 


All Registered NDE Student Participants by School, 2014—2015 


Frequency Percent 
Hilliard Elementary 42 3.7 
Briargrove Elementary 13 1.1 
Clifton Middle School 34 3.0 
Crockett Elementary School 62 5.4 
Eastwood Academy 27 2.4 
Holland Middle School 97 8.4 
Long Middle School 19 LF 
Mickey Leland College Preparatory 280 24.4 
Montgomery Elementary 107 9.3 
Pershing Middle School 57 5.0 
Piney Point 17 1.5 
Project Chrysalis MS 8 7 
Ryan Middle School 2 2 
Travis Elementary 58 5.1 
Washington High School 76 6.6 
Welch Middle School 16 1.4 
Westside High School 32 2.8 
White Elementary 81 Zl 
Young Women’s College Preparatory 120 10.5 
Total 1148 100.0 
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NDE Student Survey Sample by School 


School Frequency Percent 
Washington High School 28 7.6 
Gregory Lincoln Education Middle School 1 3 
Pershing Middle School 97 26.3 
Travis Elementary 51 13.8 
Eastwood Academy for Academic Achievement 6 1.6 
Gregory-Lincoln 3 8 
Mickey Leland College Preparatory Academy 58 15.7 
Young Women’s College Preparatory Academy 96 26.0 
Hilliard Elementary 29 7.9 
Total 369 100.0 
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Appendix C 


Propensity Score Matching Bias Analysis 


Mean t-test vty 
Variable Treated Control % bias t p>t v(C) 
Gender 31606 31606 0.0 -0.00 1.000 
Eco. Status 55959 95959 0.0 -0.00 1.000 
LEP .09845 .09845 0.0 0.00 1.000 
G/T 36788 36788 0.0 -0.00 1.000 
Special Ed. .01554 01554 0.0 -0.00 1.000 
At Risk 33161 33161 0.0 -0.00 1.000 


* if variance ratio outside [0.75; 1.33] 


Ps R2 LR chi2 — p>chi2 Mean Bias Med Bias B R % Var 


-0.000 -0.00 1.000 0.0 0.0 0.0 1.00 


* if B>25%, R outside [0.5; 2] 
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Appendix D 
Treated and Control Group Statistics 
Science DLA 
Treated (NDE) Controls 
(n = 193) (46,362) 
% Correct % Correct Difference S.E. T-stat 
Unmatched 66.8 54.3 12.5 1.502 8.30 
Matched 66.8 58.9 7.9 2.078 3.77 
Prior to Matching Treated Controls 
(n = 193) (46,362) 
% Correct % Correct 
Gender 
Male 31.6 49.2 
Female 68.4 50.8 
Economically disadvantaged 56.0 76.4 
At risk 33.2 66.9 
LEP 9.8 30.8 
Special Ed 1.6 7.8 
G/T 36.8 17.3 
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NDE Student Survey Sample Science Interest 


n Pre Std. | Post Std. t 

Read books or magazines about science 
outside of school. 369 2.0 | .8905 | 2.1 | .9411 | 2.496 .013* 
Watched news on TV that involved science 
concepts. 369 2.4 | .9541 | 2.5 | 9297 | 1.447 149 
Talked about science to parents or family 
members. 369 2.4 | 1.009 | 2.3 | .9761 | -1.500 134 
Participated in after-school science activities. 

369 1.8 | .9821 | 2.0 | 1.020 | 3.533 | .000*** 
Visited websites about science. 

369 2.3 | .9718 | 2.4 | .9817 | 1.188 .235 
Attended science study groups. 

369 1.7 | .9835 1.9 | .9974 | 3.471 | .001** 
Talked to my friends about science. 

369 2.2 | 1.004 | 2.2 | 1.004 349 727 
Participated in science competitions. 

369 1.9 | .9945 | 2.0 | 1.016 | 1.724 .086 
Talked to my teacher about science. 

369 2.5 | 1.035 | 2.6 | 1.006 | 1.4921 .137 
Had fun when learning science. 

369 3.1 | .9492 | 3.1 | .8892 -.627 ool 


Statistical significance: *p < .05; **p < .01; ***p < .001 
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NDE Student Survey Sample Science Attitude 


Appendix F 


New Digital Energy Game, 2014-2015 


Std. Std. 
n Pre | Dev. Post Dev. t p 
I am interested in learning more 3.0 
science. 368 3.1 | .8892 j 9435 -2.277 | .023* 
Science will help me become 
successful. 364 3.3 | .8723 3.2 9574 -.985 325 
I like studying science. 360 2.9 | .9701 2.8 1.007 -1.408 .160 
Science energy game will help me 
learn difficult science theories that I 
may not understand without seeing in 364 3.0 9344 28 1.015 -2.550 | .011* 
the game. , ; 
Studying science is worthwhile 
because it will improve my career 9252 .9862 -1.831 .068 
st 3.2 3.1 
opportunities 368 
Making an effort to study science is 
worth it because it will help me get 
into college. 363 3.3 | .8889 3.1 9412 -2.967 | .003** 
I study science because I know it is 9444 .9910 -1.416 .158 
3.1 3.0 
useful to me. 357 
I would like to have a career involving 
science. 363 2.7 | 1.106 2.7 1.114 321 748 


Statistical significance: *p < .05; **p < .01 
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